In recent years， anchor-based 3D hand pose estimation methods are becoming popular， and Anchor-to-Joint （A2J） is one of the more representative methods. In A2J， anchor points are densely set on depth map， and neural network is used to predict offsets between anchor points and key points together with weights of anchor points； predicted offsets and weights are used to calculate the coordinates of key points in a weighted summation mode to reduce noise in network regression results. A2J methods are simple and effective， but they are sensitive to ill-suited network structure and prone to inaccurate regression due to loss function. Therefore， an improved network HigherA2J was proposed. Firstly， a single branch jointly predicted X， Y and Z offsets between anchors and key points to better utilize 3D characteristics of depth map； secondly， network branch structure was simplified to reduce network parameters； finally， the loss function for key point estimation was designed， combined with offset estimation loss， which improved the overall estimation accuracy effectively. Experimental results show the reductions in average hand pose estimation error of 0.32 mm， 0.35 mm and 0.10 mm compared to conventional A2J on three datasets NYU， ICVL and HANDS 2017 respectively.